Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sound addon #3494

Closed
wants to merge 9 commits into from
Closed

sound addon #3494

wants to merge 9 commits into from

Conversation

jerch
Copy link
Member

@jerch jerch commented Oct 6, 2021

WIP for a sound addon.

Currently working:

  • DECPS prototype with blocking sound output (on every note currently)
  • get audio permissions through user action (needed for safari)

TODO:

  • move SoundService functionality to addon for onBell event
  • make user action for audio permission configurable
  • polish tone generator for DECPS
  • get buffer/blocking semantics from VT520 right
  • check for correct blocking timings, esp. for tiny offsets from promise code
  • clarify meaning of note=0, same for defaults in all param positions

The early DECPS functionality can be tested with the example script:

$> addons/xterm-addon-sound/fixture/example.sh

Note: If the output stops at "C5" without any sound, click somewhere on the browser background (that click is currently hardlinked as user action to get the audio permissions).

Shall fix #3316, once we get to v5.

- working prototype for DECPS sequence (with blocking sound output)
- hack to ask for sound permission from user action (needed for safari)
- partial fix of screen update issue in inputhandler (needs real fix)
@jerch jerch marked this pull request as draft October 6, 2021 18:45
@jerch jerch self-assigned this Oct 6, 2021
@j4james
Copy link

j4james commented Oct 7, 2021

This is awesome! I rather like your tone generator - its seems less harsh than the MIDI synth I'm using. And your implementation seems fairly close to mine. I ran your example.sh test script on both terminals at the same time, and it sounded to me like your volume levels are about the same as mine. The only difference (other than the buffering) was when it got to the durations part of the test - my implementation seems to be interpreting the timing somewhat faster, so it ended up finishing about a second ahead of yours. I'm not sure if that's a real issue for one of us, or just a minor difference that ended up accumulating (possibly related to the buffering). Will need to double check my code.

@j4james
Copy link

j4james commented Oct 7, 2021

I think I found one genuine bug though. My understanding is that printf "\e[4;32;0,~" should produce a second of silence, but you appear to be interpreting note 0 the same as 1, i.e a C5.

@jerch
Copy link
Member Author

jerch commented Oct 7, 2021

@j4james Thx for checking.

... my implementation seems to be interpreting the timing somewhat faster, so it ended up finishing about a second ahead of yours.

Oh I have not balanced the timings yet, there is a high chance, that the promise handling eats some microseconds here adding to a major offset in the end.

I think I found one genuine bug though. My understanding is that printf "\e[4;32;0,~" should produce a second of silence, but you appear to be interpreting note 0 the same as 1, i.e a C5.

I wasnt sure about the meaning of zero there, so I went with "zero means one". I know thats a rather bold assumption, but not totally far fetched (as many DEC sequences have that ZDM defaulting to 1 thingy). Guess this needs to be checked against the real device as well.

Added both to the TODO/reminder list.

@j4james
Copy link

j4james commented Oct 7, 2021

I wasnt sure about the meaning of zero there, so I went with "zero means one".

Again this was one of those things that wasn't clear on the page documenting the Play Sound control itself, but if you look in section 2.17 Sound, in the Set-Up chapter, it mentions that 0=silent when describing the Pnote parameter. Also in table 4.7.3 Audible Attributes Control Functions, Pnote 0 is listed as silent.

@jerch
Copy link
Member Author

jerch commented Oct 12, 2021

@j4james Fixed the "sound of silence".

About the timings - it seems my timings have like +-10ms startup, while the runtime seems to shift by ~5ms average. I think I cannot get this more accurate for 2 reasons:

  • under load the promises will schedule later, I have no way to declare RT stuff with Javascript, and promises are just a cooperative multitasking hack in JS.
  • The time resolution is ~20ms on the audio context, so even on the audio contexts internal timer things will shift by 10ms average.

Edit:
I think I can at least lower the promise callback pressure by pre-scheduling the rampDown on start already. - Yepp, works in all 3 major browser engines, changed it.

@jerch
Copy link
Member Author

jerch commented Oct 12, 2021

Thinking about proper blocking semantics - Before I can grab the device, here are some thoughts about a somewhat more useful blocking scheme than blocking on every single note:

  • There is a FIFO buffer of max. 16 notes, independent of duration and volume. (Whether the buffer is independent of duration/volume is not clear from the docs, needs device check.)
  • Blocking will only occur on every 17th note at input IO level in-band, thus with proper flow control it may halt app side. Everything else in between will not block from the sound endpoint (but ofc might still block due to other constraints from slow scroll or flowcontrol in general).
  • On finishing a note, it drops from the buffer, making room for the next one (needs device check whether it is on finishing or on start). This may unblock a 17th note.
  • Note rotation on the buffer takes no time in term of the audio playback, thus entering or dropping notes on the buffer does not influence play times or create pauses.
  • The association of a note to a sequence does not matter, thus blocking might occur within a single sequence, if a note happens to be the 17th arriving at the buffer. (Also needs device check.)

Imho with such a model advanced text updates paired with sound output gets possible. Here the only challenge for app devs will be, to anticipate the latency of screen updates to not run into full buffer drains, thus to deliver shorter notes fast enough. And ofc they may not overdo the screen updates in between and cause other blocking reasons.

@jerch
Copy link
Member Author

jerch commented Nov 16, 2021

Unlikely to happen, thus closing.

@jerch jerch closed this Nov 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bell handling
2 participants